✅ Auto-approved drewstone PR — c78df7c9

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-08T01:08:53Z}

🟡 Value Audit — sound-with-nits

💰 Value — sound-with-nits

Adds a campaign-backed runner, metric calibration, and standard retrieval-format helpers for BenchmarkAdapter implementations, fitting the existing substrate grain; a few minor naming and helper-duplication nits.

• What it does: Adds a unified benchmark runner (runBenchmarkAdapter) that wires any BenchmarkAdapter into the existing runCampaign infrastructure, produces JSON/Markdown reports with split/tag/dimension summaries and cost/latency distributions, adds a weak/strong metric calibration helper (calibrateBenchmarkMetric), and adds standard retrieval format parsers/evaluators (BEIR/MTEB/MS MARCO/TREC/MIRACL-sty
• Goals it achieves: Provides a single substrate primitive for running objective benchmarks against deterministic splits, scoring responses, tracking cost/latency, and persisting standardized reports. This lets teams add new retrieval or classification benchmarks without reimplementing campaign orchestration, report rendering, or metric math, and gives the package a first-party smoke benchmark (routing) that exercises
• Assessment: Good on merits. The change is coherent with the codebase: it reuses runCampaign, CampaignStorage, JudgeConfig, deterministicSplit, and inMemoryCampaignStorage; it follows the package's fail-loud philosophy (e.g., dataset loaders throw clear setup errors); tests cover the new runner, calibration, and retrieval scoring. The new metadata fields on BenchmarkAdapter make the contract more s
• Better / existing approach: No materially better architecture — the campaign-backed runner is the right approach. Two minor improvements: (1) the new BenchmarkReport interface in src/benchmarks/runner.ts collides with the legacy BenchmarkReport in src/types.ts (used by BenchmarkRunner/formatBenchmarkReport); rename it (e.g., BenchmarkRunReport) to avoid package-level confusion. (2) The local distribution/`per
• Model: opencode/kimi-for-coding/k2p7
• Bridge attempts: 1

🎯 Usefulness — sound-with-nits

Clean, well-integrated runner that bridges the existing BenchmarkAdapter contract to the existing runCampaign orchestrator and fills a real retrieval-metrics gap; only minor naming-overlap with the legacy BenchmarkRunner.

• Integration: Reachable via export * as benchmarks at src/index.ts:1179. runBenchmarkAdapter (src/benchmarks/runner.ts:89) wraps runCampaign (src/campaign/run-campaign.ts:128) with a correctly-shaped JudgeConfig (src/benchmarks/runner.ts:131) — type signatures for JudgeConfig/JudgeScore/Scenario/CampaignCellResult/CampaignResult all verified against src/campaign/types.ts. The routing adapter already imple
• Fit with existing patterns: Fits the codebase grain precisely: it is the missing orchestrator for the BenchmarkAdapter contract that already lived in src/benchmarks/types.ts:81. It reuses runCampaign (resumability, manifest hash, cost ceiling, concurrency, expectUsage) rather than reinventing it, and persists reports through the same CampaignStorage abstraction (src/campaign/storage.ts) that the campaign spine uses.
• Real-world viability: Concurrency, error paths, and cost ceilings are inherited from runCampaign (maxConcurrency lanes, per-cell timeout/abort, costCeilingReached short-circuit). Failed cells are excluded from the successful rows before distribution/slice aggregation and surfaced via cellsFailed (src/benchmarks/runner.ts:227). distribution() and mean() guard against empty/NaN inputs (return zeros). `firstJudg
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

💰 Value Audit

🟡 Local distribution helpers duplicate existing repo patterns [duplication] ``

src/benchmarks/runner.ts:343-369 defines its own distribution, percentile, mean, and clamp01 helpers. The codebase already has many local copies of quantile/percentile/median helpers (src/release-confidence.ts:564, src/matrix/aggregation.ts:24, src/fuzz/cube.ts:47, src/rl/dataset.ts:109, src/judge-calibration.ts:521). None of the existing helpers exactly match the BenchmarkDistribution shape (n/min/mean/median/p90/max), so reuse is not trivial, but this is another copy of a common patt

🟠 BenchmarkReport name collides with legacy BenchmarkReport [better-architecture] ``

src/benchmarks/runner.ts:39 defines BenchmarkReport, but src/types.ts:107 already exports a different BenchmarkReport used by BenchmarkRunner and formatBenchmarkReport. Both are exported from the package (the legacy one from @tangle-network/agent-eval, the new one from @tangle-network/agent-eval/benchmarks). Rename the new one (e.g., BenchmarkRunReport or BenchmarkAdapterReport) to avoid import-time confusion and maintenance mistakes.

🟡 Existing example adapters lack new metadata fields [against-grain] ``

The PR adds id, family, taskKind, description, and defaultMetric to BenchmarkAdapter and updates src/benchmarks/routing/index.ts, but the reference implementations in examples/benchmarks/gsm8k/index.ts and examples/benchmarks/swebench-lite/index.ts do not set them. This is inconsistent; consumers copying those examples will produce adapters with missing metadata, so reports will fall back to 'custom' and lose benchmark identity.

🎯 Usefulness Audit

🟡 Three overlapping 'benchmark runner' surfaces may briefly confuse consumers [problem-fit] ``

The repo now has (1) BenchmarkRunner class in src/benchmark.ts (legacy multi-turn agent scenarios via TCloud), (2) runBenchmarkAdapter in src/benchmarks/runner.ts (dataset-backed adapter orchestration via runCampaign), and (3) product-benchmark/ export helpers. Each has a distinct job and lives in a distinct namespace/path, so this is NOT a duplication — but the naming overlap (two things called 'benchmark runner') could send a new consumer to the wrong surface. Not gating; consider a one-